{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 进阶篇：循环遍历优化\n",
    "循环遍历是所有编程语言的基础语法，初学者为了快速实现功能，依懒性较强。但如果从运算时间性能上考虑可能不是特别好的选择。\n",
    "本节将介绍几个常见的提速方法，一个比一个快，了解pandas本质，才能知道如何提速。\n",
    "下面是一个例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from zyf_timer import timeit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date_time</th>\n",
       "      <th>energy_kwh</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2013-01-01 00:00:00</td>\n",
       "      <td>0.586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2013-01-01 01:00:00</td>\n",
       "      <td>0.580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2013-01-01 02:00:00</td>\n",
       "      <td>0.572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2013-01-01 03:00:00</td>\n",
       "      <td>0.596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2013-01-01 04:00:00</td>\n",
       "      <td>0.592</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            date_time  energy_kwh\n",
       "0 2013-01-01 00:00:00       0.586\n",
       "1 2013-01-01 01:00:00       0.580\n",
       "2 2013-01-01 02:00:00       0.572\n",
       "3 2013-01-01 03:00:00       0.596\n",
       "4 2013-01-01 04:00:00       0.592"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/demand_profile.csv', parse_dates=['date_time'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于上面的数据，我们现在要增加一个新的特征，但这个新的特征是基于一些时间条件生成的，根据时长（小时）而变化，如下：\n",
    "\n",
    "| Traiff   | Centsper kWh | Time Range     |\n",
    "| -------- | ------------ | -------------- |\n",
    "| Peak     | 28           | 17:00 to 24:00 |\n",
    "| Shoulder | 20           | 7:00 to 17:00  |\n",
    "| Off-Peak | 12           | 0:00 to 7:00   |\n",
    "\n",
    "因此，如果你不知道如何提速，那正常第一想法可能就是用`apply`方法写一个函数，函数里面写好时间条件的逻辑代码。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_tariff(kwh, hour):\n",
    "    \"\"\"计算每个小时的电费\"\"\"    \n",
    "    if 0 <= hour < 7:\n",
    "        rate = 12\n",
    "    elif 7 <= hour < 17:\n",
    "        rate = 20\n",
    "    elif 17 <= hour < 24:\n",
    "        rate = 28\n",
    "    else:\n",
    "        raise ValueError(f'Invalid hour: {hour}')\n",
    "    return rate * kwh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后使用for循环来遍历df，根据apply函数逻辑添加新的特征，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "花费时间:  13.3903 secs\n"
     ]
    }
   ],
   "source": [
    "@timeit\n",
    "def apply_tariff_loop(df):\n",
    "    \"\"\"用for循环计算enery cost，并添加到列表\"\"\"\n",
    "    energy_cost_list = []\n",
    "    \n",
    "    for i in range(len(df)):\n",
    "         # 获取用电量和时间（小时）.\n",
    "        energy_used = df.iloc[i]['energy_kwh']\n",
    "        hour = df.iloc[i]['date_time'].hour\n",
    "        energy_cost = apply_tariff(energy_used, hour)\n",
    "        energy_cost_list.append(energy_cost)\n",
    "    \n",
    "    df['cost_cents'] = energy_cost_list\n",
    "\n",
    "apply_tariff_loop(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于那些写Pythonic风格的人来说，这个设计看起来很自然。然而，这个循环将会严重影响效率。原因有几个：\n",
    "- 首先，它需要初始化一个将记录输出的列表。\n",
    "- 其次，它使用不透明对象范围(0，len(df))循环，然后再应用apply_tariff()之后，它必须将结果附加到用于创建新DataFrame列的列表中。另外，还使用df.iloc [i]['date_time']执行所谓的链式索引，这通常会导致意外的结果。\n",
    "\n",
    "这种方法的最大问题是计算的时间成本。对于8760行数据，此循环花费了13秒钟。\n",
    "\n",
    "接下来，一起看下优化的提速方案。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、使用 iterrows循环\n",
    "第一种可以通过pandas引入iterrows方法让效率更高。这些都是一次产生一行的生成器方法，类似scrapy中使用的yield用法。\n",
    "\n",
    ".itertuples为每一行产生一个namedtuple，并且行的索引值作为元组的第一个元素。nametuple是Python的collections模块中的一种数据结构，其行为类似于Python元组，但具有可通过属性查找访问的字段。\n",
    "\n",
    ".iterrows为DataFrame中的每一行产生（index，series）这样的元组。\n",
    "\n",
    "在这个例子中使用.iterrows，我们看看这使用iterrows后效果如何。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "花费时间:  3.9268 secs\n"
     ]
    }
   ],
   "source": [
    "@timeit\n",
    "def apply_tariff_loop(df):\n",
    "    \"\"\"用for循环计算enery cost，并添加到列表\"\"\"\n",
    "    energy_cost_list = []\n",
    "    \n",
    "    for index, row in df.iterrows():\n",
    "        # 获取用电量和时间（小时）\n",
    "        energy_used = row['energy_kwh']\n",
    "        hour = row['date_time'].hour\n",
    "        energy_cost = apply_tariff(energy_used, hour)\n",
    "        energy_cost_list.append(energy_cost)\n",
    "    \n",
    "    df['cost_cents'] = energy_cost_list\n",
    "\n",
    "apply_tariff_loop(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这样的语法更明确，并且行值引用中的混乱更少，因此它更具可读性。\n",
    "\n",
    "时间成本方面：快了3倍多！\n",
    "\n",
    "但是，还有更多的改进空间，理想情况是可以用pandas内置更快的方法完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、pandas的apply方法\n",
    "我们可以使用.apply方法而不是.iterrows进一步改进此操作。pandas的.apply方法接受函数callables并沿DataFrame的轴(所有行或所有列)应用。下面代码中，lambda函数将两列数据传递给apply_tariff()："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "花费时间:  0.8535 secs\n"
     ]
    }
   ],
   "source": [
    "@timeit\n",
    "def apply_tariff_loop(df):\n",
    "    df['cost_cents'] = df.apply(lambda row: apply_tariff(kwh=row['energy_kwh'], hour=row['date_time'].hour), axis=1)\n",
    "\n",
    "apply_tariff_loop(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "apply的语法优点很明显，行数少，代码可读性高。在这种情况下，所花费的时间大约是iterrows方法的五分之一。\n",
    "\n",
    "但是，这还不是“非常快”。一个原因是apply()将在内部尝试循环遍历Cython迭代器。但是在这种情况下，传递的lambda不是可以在Cython中处理的东西，因此它在Python中调用并不是那么快。\n",
    "\n",
    "如果我们使用apply()方法获取10年的小时数据，那么将需要大约15分钟的处理时间。如果这个计算只是大规模计算的一小部分，那么真的应该提速了。这也就是矢量化操作派上用场的地方。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、矢量化操作：使用.isin选择数据\n",
    "### 什么是矢量化操作？\n",
    "如果你不基于一些条件，而是可以在一行代码中将所有电力消耗数据应用于该价格：df ['energy_kwh'] * 28，类似这种。那么这个特定的操作就是矢量化操作的一个例子，它是在pandas中执行的最快方法。\n",
    "\n",
    "### 但是如何将条件计算应用为pandas中的矢量化运算？\n",
    "\n",
    "一个技巧是：根据你的条件，选择和分组DataFrame，然后对每个选定的组应用矢量化操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在下面代码中，我们将看到如何使用pandas的.isin()方法选择行，然后在矢量化操作中实现新特征的添加。在执行此操作之前，如果将date_time列设置为DataFrame的索引，会更方便："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将date_time列设置为DataFrame的索引\n",
    "df.set_index('date_time', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "@timeit\n",
    "def apply_tariff_isin(df):\n",
    "    # 定义小时范围Boolean数组\n",
    "    peak_hours = df.index.hour.isin(range(17, 24))\n",
    "    shoulder_hours = df.index.hour.isin(range(7, 17))\n",
    "    off_peak_hours = df.index.hour.isin(range(0, 7))\n",
    "\n",
    "    # 使用上面apply_traffic函数中的定义\n",
    "    df.loc[peak_hours, 'cost_cents'] = df.loc[peak_hours, 'energy_kwh'] * 28\n",
    "    df.loc[shoulder_hours,'cost_cents'] = df.loc[shoulder_hours, 'energy_kwh'] * 20\n",
    "    df.loc[off_peak_hours,'cost_cents'] = df.loc[off_peak_hours, 'energy_kwh'] * 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们来看一下结果如何。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "花费时间:  0.6745 secs\n"
     ]
    }
   ],
   "source": [
    "apply_tariff_isin(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提示，上面.isin()方法返回的是一个布尔值数组，如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[False, False, False, ..., True, True, True]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "布尔值标识了DataFrame索引datetimes是否落在了指定的小时范围内。然后把这些布尔数组传递给DataFrame的.loc，将获得一个与这些小时匹配的DataFrame切片。然后再将切片乘以适当的费率，这就是一种快速的矢量化操作了。\n",
    "\n",
    "上面的方法完全取代了我们最开始自定义的函数apply_tariff()，代码大大减少，同时速度起飞。\n",
    "\n",
    "运行时间比apply快几十倍！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 四、还能更快？\n",
    "太刺激了，我们继续加速。\n",
    "\n",
    "在上面apply_tariff_isin中，我们通过调用df.loc和df.index.hour.isin三次来进行一些手动调整。如果我们有更精细的时间范围，你可能会说这个解决方案是不可扩展的。但在这种情况下，我们可以使用pandas的pd.cut()函数来自动完成切割："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "@timeit\n",
    "def apply_tariff_cut(df):\n",
    "    cents_per_kwh = pd.cut(x=df.index.hour,\n",
    "                           bins=[0, 7, 17, 24],\n",
    "                           include_lowest=True,\n",
    "                           labels=[12, 20, 28]).astype(int)\n",
    "    df['cost_cents'] = cents_per_kwh * df['energy_kwh']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面代码pd.cut()会根据bin列表应用分组。\n",
    "\n",
    "其中include_lowest参数表示第一个间隔是否应该是包含左边的。\n",
    "\n",
    "这是一种完全矢量化的方法，它在时间方面是最快的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "花费时间:  0.0160 secs\n"
     ]
    }
   ],
   "source": [
    "apply_tariff_cut(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到目前为止，使用pandas处理的时间上基本快达到极限了！只需要花费不到一秒的时间即可处理完整的10年的小时数据集。\n",
    "但是，最后一个其它选择，就是使用 NumPy，还可以更快！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 五、使用Numpy继续加速\n",
    "使用pandas时不应忘记的一点是Pandas的Series和DataFrames是在NumPy库之上设计的。并且，pandas可以与NumPy阵列和操作无缝衔接。\n",
    "下面我们使用NumPy的 digitize()函数更进一步。它类似于上面pandas的cut()，因为数据将被分箱，但这次它将由一个索引数组表示，这些索引表示每小时所属的bin。然后将这些索引应用于价格数组："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "@timeit\n",
    "def apply_tariff_digitize(df):\n",
    "    prices = np.array([12, 20, 28])\n",
    "    bins = np.digitize(df.index.hour.values, bins=[7, 17, 24])\n",
    "    df['cost_cents'] = prices[bins] * df['energy_kwh'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与cut函数一样，这种语法非常简洁易读。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "花费时间:  0.0060 secs\n"
     ]
    }
   ],
   "source": [
    "apply_tariff_digitize(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "执行速度上，仍然有提升，但是这种提升已经意义不大了。不如将更多精力去思考其他的事情。\n",
    "\n",
    "Pandas可以提供很多批量处理数据方法的备用选项，这些已经在上边都一一演示过了。这里将最快到最慢的方法排序如下：\n",
    "\n",
    "1. 使用向量化操作：没有for遍历的Pandas方法和函数。\n",
    "\n",
    "2. 使用.apply()方法。\n",
    "\n",
    "3. 使用.itertuples()：将DataFrame行作为nametuple类从Python的collections模块中进行迭代。\n",
    "\n",
    "4. 使用.iterrows()：将DataFrame行作为(index，pd.Series)元组数组进行迭代。虽然Pandas的Series是一种灵活的数据结构，但将每一行生成一个Series并且访问它，仍然是一个比较大的开销。\n",
    "\n",
    "5. 对逐个元素进行循环，使用df.loc或者df.iloc对每个单元格或者行进行处理。\n",
    "\n",
    "【注】以上顺序不是我的建议，而是Pandas核心开发人员给的建议。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
